Testing Mixture Models of Transitive Preference: Comments on Regenwetter, Dana, and Davis-stober (2011) I Thank

This paper contrasts two approaches to analyzing transitivity of preference and other behavioral properties in choice data. The approach of Regenwetter, Dana, and Davis-Stober (2011) assumes that on each choice, a decision maker samples randomly from a mixture of preference orders in order to determine whether A is preferred to B. In contrast, Birnbaum and Gutierrez (2007) assumed that within each block of trials the decision maker has a “true” set of preferences, and random “errors” generate variability of response. In this latter approach, preferences are allowed to differ between people; within-person, they might differ between repetition blocks. Both approaches allow mixtures of preferences, both assume a type of independence, and both yield statistical tests. They differ with respect to the locus of independence in the data. The approaches also differ in the criterion for assessing the success of the models. Regenwetter et al. (2011) fit only marginal choice proportions and assumed choices are independent, which means that a mixture cannot be identified from the data. Birnbaum and Gutierrez fit choice combinations with replications; their approach allows estimation of the probabilities in the mixture. It is suggested that we should separate tests of the stochastic model from the test of transitivity. Evidence testing independence and stationarity assumptions are presented. Available data appear to fit the assumption that errors are independent better than they fit the assumption that choices are independent. Transitivity of Preference 3 Regenwetter, Dana, and Davis-Stober (2011) presented a theoretical analysis, reanalysis of published evidence, and a new experiment to argue that preferences are transitive in a situation that was previously theorized to produce systematic violations of transitivity. Tversky (1969) argued that some participants use a lexicographic semiorder to compare gambles and this process led them to systematically prefer A over B, B over C, and C over A. Regenwetter, et al. reanalyzed Tversky’s (1969) data and concluded that they do not refute a mixture model in which each person on each trial might use a different transitive order to determine her or his preferences. In this note, I will contrast their approach with a similar one that my collaborators and I have been using recently. I will provide arguments and evidence against the method of analysis advocated by Regenwetter et al. (2011). Morrison (1963) reviewed both weak stochastic transitivity (WST) and the triangle inequality (TI) as properties implied by various models of paired comparisons. He argued that both properties should be analyzed. Tversky (1969) cited Morrison but reported only tests of WST. Regenwetter, et al. reanalyzed Tversky’s data and showed that violations of the TI are not “significant” for Tversky’s data, according to a new statistical test. They argued in favor of mixture models that can be tested via marginal (binary) choice proportions and concluded that these mixture models are compatible with published evidence in the literature and with results of a new experiment. Although I agree with much of what Regenwetter, et al. said concerning previous literature, including the Iverson and Falmagne (1985) reanalysis of Tversky (1969), and I agree with their conclusion that evidence against transitivity is underwhelming, I will review points of disagreement between their approach and one that I prefer. The Problem of Using Marginal Choice Proportions Transitivity of Preference 4 I agree with Regenwetter, et al.’s criticism of WST, which is the assumption that if p(AB) > 1⁄2, and p(BC) > 1⁄2, then p(AC) > 1⁄2, where p(AB) represents the probability of choices in which B is preferred to A. As has been noted by them and others, if a given person has a mixture of transitive orders, WST can be violated even when every response pattern is transitive. Let us consider an experiment in which a participant is asked to make all pairwise comparisons of three stimuli, A, B, and C. Suppose these three choices are presented intermixed among filler choices in blocks of trials, and each choice appears once in each of 100 blocks. Each block contains all three choices and is called a repetition. In Table 1, 0 represents preference for the first item listed in each choice and 1 represents preference for the second item; A f B denotes A is preferred to B. Note that in Example 1, only three transitive patterns have nonzero frequency. In 33 repetitions, the person preferred A f B, B f C and A f C (pattern 000); 33 times this person chose C f A, A f B and C f B (pattern 011); and in 34 cases, this person chose B f C, C f A and B f A (pattern 101). When we aggregate across data patterns, however, WST is violated in the marginal proportions, P(AB) = .66, P(BC) = .67, and P(CA) = .67, and given enough data, such findings allow one to reject the hypothesis that the corresponding binary choice probabilities satisfy WST. By combining across data patterns and using WST, we might reach the wrong conclusion that this person was violating transitivity, when in fact, the person has a mixture of transitive choice patterns. Insert Table 1 about here According to the triangle inequality, 0 ≤ p(AB) + p(BC) p(AC) ≤ 1. In this case, the sum of the corresponding binary choice proportions is 1 (i.e, .66 + .67 .33 = 1), so the TI condition is satisfied by the proportions and therefore we cannot reject the hypothesis that Transitivity of Preference 5 this relation holds for the corresponding probabilities. Thus, the TI correctly diagnosed Example 1 as compatible with a mixture of transitive patterns. For this reason, Regenwetter et al. (2011) argue that we should use the TI to determine if transitivity is acceptable, rather than WST. However, it is easy to construct examples in which every response pattern is intransitive and the TI is satisfied. In Example 2 of Table 1 the TI is satisfied and the marginal choice proportions are virtually the same as in Example 1; however, these data are perfectly intransitive. Example 3 shows that TI can also be satisfied when there is a mixture of transitive and intransitive patterns. (Note that if we know the distribution over preference patterns, we can compute the marginal choice probabilities (e.g., p(AB) = p(000) + p(001) + p(010) + p(011), but we cannot use binary choice probabilities to identify the relative frequencies of response patterns.) The moral I draw from these examples and others is that we should be analyzing data patterns rather than marginal choice proportions. In my opinion, Regenwetter et al. (2011) have not gone far enough in their criticism of WST by extending their criticism to other properties like the TI that are defined on binary choice proportions. Unfortunately, the Tversky (1969) data have not been saved in a form that allows us any longer to analyze them as in Table 1. From marginal choice proportions alone, it is not possible to know if his data resembled Examples 1, 2, or 3. Regenwetter, et al. (2010) considered the possibility of examining data as in Table 1, but concluded that it would require more extensive experiments than have yet been done on this issue. In the next section, two rival stochastic models for such data are presented. Both Transitivity of Preference 6 allow for a mixture of mental states; they both lead to statistical tests, but they differ with respect to the locus of the independence assumptions. Two Stochastic Models of Choice Combinations Random Utility Mixture Model: Independent Choices As noted by Regenwetter, et al. (2011), the term “random utility model” has been used in different ways in the literature. Further, the term “mixture model” will not distinguish two approaches I compare here. I use the term “random utility mixture model (RUMM)” here to refer to the model and statistical independence assumptions in Regenwetter et al. (2011). They included filler items between choices and arranged their study so that a participant could not review or revise his or her previous choices, based on the theory that these precautions would make the data satisfy independence and stationarity (Regenwetter, Dana, & Davis-Stober, 2010). I will focus on two types of independence that are assumed in this approach that I find empirically doubtful: First, responses to the same item presented twice in different trial blocks (separated by filler trials) should be independent. That is, when presented twice with the same item, response to the second presentation should be independent of the response to the first. Second, responses to related items, separated by fillers, should be independent; that is, when choosing between A and B, the probability to choose A should be independent of the response given in the choice between A and C. In addition, the statistical assumption of “ iid = independent and identically distributed” implies that the probability to choose A over B does not change systematically over trials during the course of the study; I use the term “stationarity” to refer to this latter assumption. Transitivity of Preference 7 Regenwetter et al. (2010, 2011) did not test the effects of the filler trials nor did they test the assumptions of independence and stationarity; they fit their model to binary choice proportions. They noted that RUMM together with its statistical assumptions can be tested, but that model is not identifiable; that is, we cannot identify the distribution over preference orders that a person might have in her or his mental set. In other words, when the transitive model fits, there are many possible mixtures of preference orders that might account for a given set of binary choice proportions. Table 2 shows hypothetical data for a case in which three stimuli have been presented for comparison on 200 repetitions. The marginal choice proportions are P(AB) = 0.795, P(BC) = 0.600, and P(AC) = 0.595; they satisfy the TI. Therefore, these data satisfy the transitive RUMM, according to the methods advocated by Regenwetter, et al. (2010, 2011). However, an analysis of response patterns, as shown below, leads to very different conclusions. Insert Table 2 about here. In RUMM, the theoretical probability that a person chooses A over B is the probability of the union of all preference patterns in which A f B. Because the patterns in Table 2 are mutually exclusive, we can sum probabilities over all patterns in which 0 appears in the first position in Table 2 (i.e., for which A f B); the theoretical probability to prefer A over B is given as follows: ! pAB = p000 + p001 + p010 + p011 (1) Equation 1 is a bit more general than Equation 5 of Regenwetter et al. (2011), who do not consider intransitive preferences to be allowable; this expression is the general case, and a special case in which ! p 001 = p 110 = 0 is called the transitive special case). Transitivity of Preference 8 Assuming independence, the probability of any particular preference pattern is the product of the probabilities of the individual terms; for example, the predicted probability of the 001 (intransitive) preference pattern is given as follows: ! p(001) = pAB pBC (1" pAC ) (2)